METHOD, APPARATUS, AND RECORDING MEDIUM FOR IMAGE PROCESSING 

BACKGROUND OF THE INVENTION 
Field of the Invention 
5 The present invention relates to an image processing 

method and an image processing apparatus for carrying out 
color balance adjustment processing on an image obtained by 
using a digital camera when the image is output from a printer 
or the like, and also relates to a computer-readable recording 
10 medium storing a program to cause a computer to execute the 
image processing method. 

Description of the Related Art 
In a digital electronic still camera (hereinafter called 
a digital camera), an image obtained by photographing is 
15 recorded as digital image data in a recording medium such as 
an IC card or a built-in memory installed in the digital camera . 
Based on the digital image data having been recorded, the 
photographed image can be reproduced as a hard copy such as 
a print or a soft copy on a display device. When an image 
20 obtained by a digital camera is reproduced as has been 
described above, the image is expected to have as high' a 
quality as a photograph printed from a negative film. 

Meanwhile, human vision has a mechanism called chromatic 
adaptation for perceiving white paper as white not as bluish 
25 or reddish even in daylight or in light from a fluorescent 
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lamp or a tungsten light each having a different spectrum 
energy distribution from normal daylight. Taking such 
chromatic adaptation into consideration, a digital camera 
having a function (an AWB function) has been proposed for 
5 carrying out automatic white balance adjustment processing 
(AWB processing) in accordance with a light source used at 
the time of photographing on original image data representing 
a photographed image so as to obtain an image not depending 
on a color of the light source such as tungsten light, light 

10 from a fluorescent lamp, or daylight. 

Furthermore, a digital camera for outputting image data 
after image processing such as chroma enhancing processing 
on the image data so as to cause a color to become adequate 
when displayed on a CRT monitor has been known, since image 

15 data obtained by a digital camera are frequently displayed 

on a CRT monitor. 

Meanwhile, when image data obtained by a digital camera 
are output as a hard copy from a printer or the like, the 
printer carries out AWB processing adequate for printing on 

20 the image data and the image data after the processing are 
printed . 

Due to limitation caused by processing speed or hardware, 
the AWB processing carried out in a digital camera is 
comparatively simple processing for adjusting a gain of each 
25 color by equalizing averages of RGB colors, assuming an 



2 



average of all pixel values being gray. Meanwhile, a printer 
can carry out more sophisticated AWB processing than that of 
a digital camera. For example, as has been described in 
Japanese Patent Application No. 11 ( 1999 ) -70186 , RGB values 
5 of each pixel are converted into chromaticity values to find 
averages of the chromaticity values. The averages are then 
converted into physiological primary colors and assumed to 
be a neutral point of a light source used at the time of 
photographing. The chromaticity values are converted so as 

10 to cause the physiological primary colors to be equal to 
physiological primary colors of a reference light source for 
photographing. The converted chromaticity values are then 
converted into RGB signals. 

However, since image data output from a digital camera 

15 have been subjected to AWB processing, colors look different 
from colors in an original image. Therefore, for image data 
obtained by a digital camera, the function of sophisticated 
AWB processing carried out by a printer is not fully used. 
In the case where the digital camera carries out image 

20 processing such as chroma enhancing processing as well as the 
AWB processing, image data obtained by the digital camera have 
distorted color balance. As a result, regardless of 
sophisticated AWB processing carried out on the image data, 
an adequate neutral point cannot be set and a high quality 

25 image cannot be obtained. 
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The present invention has been conceived based on 
consideration of the above problems . An object of the present 
invention is therefore to provide an image processing method 
and an image processing apparatus for carrying out adequate 
5 color balance adjustment processing such as AWB processing 
on image data obtained by a digital camera, and also to provide 
a computer-readable recording medium storing a program to 
cause a computer to execute the image processing method. 

SUMMARY OF THE INVENTION 

10 A first image processing method of the present invention 

is a method of obtaining color-balance adjusted image data 
by carrying out color balance adjustment processing on 
white-balance adjusted image data obtained by carrying out, 
on original image data, white balance adjustment processing 

15 using gain adjustment in a color space in which the original 
image data have been obtained. The image processing method 
comprises the steps of: 

obtaining inverse white-balance adjusted image data by 
carrying out, on the white-balance adjusted image data, 

20 inverse white balance adjustment processing which is inverse 
processing of the white balance adjustment processing; and 
obtaining the color-balance adjusted image data by 
carrying out the color balance adjustment processing on the 
inverse white-balance adjusted image data. 

25 The "original image data" refer to image data before 



4 



the white balance adjustment processing. Not only image data 
immediately after photographing by a digital camera but also 
image data after any processing, such as image data after 
standardization processing, can be used as the original image 
5 data, as long as the white balance adjustment processing has 
not been carried out. 

The "color space in which the original image data have 
been obtained" means a color space of a sensor such as CCDs 
installed in the digital camera used for obtaining the 

10 original image data. For example, an RGB color space or a 
CMY color space can be used. 

The "white balance adjustment processing using gain 
adjustment in a color space in which the original image data 
have been obtained" refers to processing for adjusting a gain 

15 of each color in the color space of the sensor for obtaining 
the original image data such as the RGB color space and the 
CMY color space so that a difference between neutral points 
depending on a light source used upon viewing the original 
image data and on a light source used at the time of 

20 photographing can be corrected. 

The "color balance adjustment processing" refers to 
processing for correcting the difference between the neutral 
points depending on the light source used at the time of 
photographing an image and viewing the image. For example, 

25 comparatively sophisticated processing such as the 
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processing described in Japanese Patent Application No. 
11 ( 1999 ) -70186 is an example of the color balance adjustment 
processing - 

The " inverse white balance adjustment processing which 
5 is inverse processing of the white balance adjustment 
processing" means processing for restoring the white-balance 
adjusted image data to the original image data. Precisely 
speaking,, the "inverse white-balance adjusted image data" 
obtained by carrying out the inverse white balance adjustment 

10 processing are not exactly the same as the original image data 
immediately after photographing , due to a calculation error 
or clipping. However, the inverse white-balance adjusted 
image data can be regarded as the original image data 
immediately after photographing. 

15 In the first image processing method of the present 

invention, it is preferable for the inverse white balance 
adjustment processing to be carried out on the white-balance 
adjusted image data, based on information when carrying out 
the white balance adjustment processing. 

20 As the "information when carrying out the white balance 

adjustment processing", parameters such as the gain of each 
color adjusted during the white balance processing can be used, 
for example. 

As the "predetermined image processing", color 
25 conversion processing for enhancing chroma or density 
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conversion processing for converting density of a scene can 
be used, for example. 

A second image processing method of the present 
invention is an image processing method for obtaining 
5 color-balance adjusted image data by carrying out color 
balance adjustment processing on white-balance adjusted 
image data obtained by carrying out white balance adjustment 
processing on original image data by using gain adjustment 
in a color space in which the original image data have been 

10 obtained and by carrying out predetermined image processing 
thereon. The image processing method comprises the steps of: 
obtaining inverse white-balance adjusted image data by 
carrying out, on the white-balance adjusted image data, 
inverse image processing which is inverse processing of the 

15 predetermined image processing and inverse white balance 
adjustment processing which is inverse processing of the white 
balance adjustment processing; and 

obtaining the color-balance adjusted image data by 
carrying out the color balance adjustment processing on the 

20 inverse white-balance adjusted image data. 

In the second image processing method of the present 
invention, it is preferable for the inverse white balance 
adjustment processing to be carried out on the white-balance 
adjusted image data, based on information when carrying out 

25 the white balance adjustment processing. 
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In the second image processing method of the present 
invention, it is preferable for the inverse image processing 
to be carried out on the white-balance adjusted image data, 
based on information when carrying out the predetermined image 
5 processing. 

As the "information when carrying out the predetermined 
image processing", information representing the amount of 
colors or tones changed by the predetermined image processing 
can be used. More specifically, in the case where the 
10 predetermined image processing is density conversion 
processing or color conversion processing for enhancing 
chroma, information representing the amount of chroma or 
density having been changed through the processing can be 
used. 

15 A first image processing apparatus of the present 

invention is an apparatus for obtaining color-balance 
adjusted image data by carrying out color balance adjustment 
processing on white-balance adjusted image data obtained by 
carrying out, on original image data, white balance adjustment 

20 processing using gain adjustment in a color space in which 
the original image data have been obtained. The image 
processing apparatus comprises: 

inverse processing means for obtaining inverse 
white-balance adjusted image data by carrying out, on the 

25 white-balance adjusted image data, inverse white balance 
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adjustment processing which is inverse processing of the white 
balance adjustment processing; and 

color balance adjustment processing means for obtaining 
the color-balance adjusted image data by carrying out the 
5 color balance adjustment processing on the inverse white- 
balance adjusted image data. 

In the first image processing apparatus of the present 
invention, it is preferable for the inverse processing means 
to carry out the inverse white balance adjustment processing 

10 on the white-balance adjusted image data , based on information 

when carrying out the white balance adjustment processing. 

A second image processing apparatus of the present 
invention is an apparatus for obtaining color-balance 
adjusted image data by carrying out color balance adjustment 

15 processing on white-balance adjusted image data obtained by 
carrying out, on original image data, white balance adjustment 
processing using gain adjustment in a color space in which 
the original image data have been obtained and by carrying 
out predetermined image processing thereon. The image 

20 processing apparatus comprises: 

inverse processing means for obtaining inverse 
white-balance adjusted image data by carrying out, on the 
white-balance adjusted image data, inverse image processing 
which is inverse processing of the predetermined image 

25 processing and inverse white balance adjustment processing 
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which is inverse processing of the white balance adjustment 
processing; and 

color balance adjustment processing means for obtaining 
the color-balance adjusted image data by carrying out the 
5 color balance adjustment processing on the inverse white- 
balance adjusted image data. 

In the second image processing apparatus of the present 
invention, it is preferable for the inverse processing means 
to carry out the inverse white balance adjustment processing 
10 on the white-balance adjusted image data, based on information 
when carrying out the white balance adjustment processing. 

Furthermore, in the second image processing apparatus 
of the present invention, it is preferable for the inverse 
processing means to carry out the inverse image processing 
15 on the white-balance adjusted image data, based on information 
when carrying out the predetermined image processing. 

The first image processing method and the second image 
processing method of the present invention may be provided 
as programs stored in a computer-readable recording medium 
20 to cause a computer to execute the methods. 

Furthermore, the first image processing apparatus and 
the second image processing apparatus may be installed in an 
output apparatus such as a printer. 

According to the present invention, the inverse 
25 white-balance adjusted image data are obtained by carrying 
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out the inverse white balance adjustment processing on the 
white-balance adjusted image data, and the inverse white- 
balance adjusted image data are subjected to the color balance 
adjustment processing. Therefore, the color-balance 
5 adjusted image data obtained in this manner can be regarded 
as image data having a color balance almost the same as the 
color balance obtained by carrying out color balance 
adjustment processing directly on the original image data. 
Consequently, the color-balance adjusted image data 

10 sufficiently reflecting processing performance of color 
balance adjustment processing can be obtained without being 
affected by white balance adjustment processing. 

In the case where the white-balance adjusted image data 
have been subjected to the white balance adjustment processing 

15 and the predetermined image processing, the inverse 
white-balance adjusted image data are obtained by carrying 
out the inverse image processing which is the inverse 
processing of the predetermined image processing and the 
inverse white balance adjustment processing on the white- 

20 balance adjusted image data. By carrying out the color 
balance adjustment processing on the inverse white-balance 
adjusted image data, the color-balance adjusted image data 
are obtained. Therefore, the color-balance adjusted image 
data obtained in this manner have been subjected to color 

25 balance adjustment processing almost the same as in the case 
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of carrying out direct color balance processing on the 
original image data. Consequently, process ing can be carried 
out on inverse white-balance adjusted image data which can 
be regarded as the original image data and have less affected 
5 color balance than in the case of carrying out the color 
balance adjustment processing on the white-balance adjusted 
image data. In this manner, the color-balance adjusted image 
data sufficiently reflecting processing performance of the 
color balance adjustment processing can be obtained by 

10 adequately setting a neutral point, without being affected 
by the white balance adjustment processing and by the 
predetermined image processing. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a block diagram showing an outline 

15 configuration of an image output system adopting an image 
processing apparatus according to an embodiment of the present 
invention; 

Figure 2 is a flow chart showing processing carried out 
in the embodiment; 
20 Figure 3 is a block diagram showing an outline 

configuration of an image output system according to another 
embodiment of the present invention; 

Figure 4 is a block diagram showing an outline 
configuration of AWB means for carrying out another kind of 
25 AWB processing; 
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Figure 5 is a diagram explaining how to generate 
processing parameters in an LMS color space (part 1); and 

Figure 6 is a diagram explaining how to generate the 
processing parameters in the LMS color space (part 2). 
5 DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Hereinafter, embodiments of the present invention will 
be explained with reference to the accompanying drawings. 

Figure 1 is a block diagram showing an outline 
configuration of an image output system adopting an image 
10 processing apparatus according to an embodiment of the present 
invention. As shown in Figure 1, the image output system in 
this embodiment generates a print by inputting output image 
data S2 obtained by a digital camera 1 to a printer 2 . 

The digital camera 1 comprises imaging means 3 for 
15 obtaining original image data SO by photographing a scene, 

first AWB means 4 for obtaining AWB image data SI by carrying 
out first AWB processing on the original image data SO obtained 
by the imaging means 3, and processing means 5 for obtaining 
the output image data S2 by carrying out image processing on 
20 the AWB image data SI. The output image data S2 are recorded 
in a recording medium 6 such as a memory stick or a memory 
card. 

The printer 2 comprises reading means 7 for reading the 
output image data S2 from the recording medium 6, inverse 
25 processing means 8 for obtaining inverse-processed image data 
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SI' by carrying out inverse image processing which is inverse 
processing of the processing carried out by the processing 
means 5 in the digital camera 1, inverse AWB means 9 for 
obtaining inverse AWB image data S0 r by carrying out, on the 
5 inverse-processed image data SI', inverse AWB processing 
which is inverse processing of the AWB processing carried out 
by the first AWB means 4, second AWB means 10 for obtaining 
processed image data S3 by carrying out second AWB processing 
different from the first AWB processing on the inverse AWB 

10 image data SO', and output means 11 for outputting the 

processed image data S3 as a hard copy. 

The imaging means 3 comprises means necessary for 
photographing a scene, such as a lens, RGB filters, and CCDs. 
The image data SO obtained by the imaging means 3 comprise 

15 3 signal values R0 , GO, and BO. Let spectral reflectivity 
of a scene be A( A ) , and let filter characteristics of the 
RGB filters in the imaging means 3 be r ( A ) , g( A ) , and b ( A ) . 
The signal values R0 , GO and BO are expressed by Equation (1) 
below: 

20 

R0=J A (A) r (A) dA 
(i) GO = J A (A) g (A) dA 

B0=J A (A) b (A) dA 

25 In the first AWB means 4, parameters a and (3 to be 
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multiplied with integrals of the signal values RO and BO are 
determined so as to equalize the integrals of the signal values 
RO , GO and BO in an entire image area represented by the 
original image data S0 f as shown in Equation (2) below: 

5 

(2) af RO - f G0-/3r BO 

v ' J entire image J entire image J entire image 

Signal values Rl , Gl, and Bl comprising the AWB image data 
SI are then found according to the following Equation (3): 

10 Rl = aR0 

G1=G0 (3) 

B1 = /?B0 

In the processing means 5, image processing comprising 
scene density conversion processing and chroma enhancement 
15 processing is carried out to generate the output image data 
S2. The scene density conversion processing is processing 
for converting the signal values Rl , Gl and Bl comprising the 
AWB image data SI into signal values R2 , G2 , and B2 which are 
RGB signal values for reproduction on a monitor or the like. 
20 More specifically, the signal value R2 can be found according 
to the following Equation (4) below: 

R2 = l . 099Rl°* 45 -0 . 099 (if Rl^0.018) 
R2 = 4.50R1 (if RK0.018) 

Likewise, the signal values G2 and B2 can also be found by 
25 the above operation replacing R with G or B. 
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The chroma enhancement processing converts the signal 
values R2, G2 , and B2 obtained by the scene density conversion 
processing into YCC luminance color-difference components as 
shown in Equation (5) below, and obtains a component Cb' and 
5 a component Cr' by multiplying a component Cb and a component 
Cr obtained according to Equation (5) with parameters a and 
b, as shown in Equation (6) below: 
Y=0 . 2 99R2+0 . 58 7G2+0 . 114B2 
Cb=(B2-Y)/1.772 (5) 
10 Cr=(R2-Y) /l .402 

Cb'=aXCb+0.5 

Cr'=bXCr+0.5 (6) 
Since -0.5^ Cb ^0.5 and -0.5^ Cr ^0.5, 0.5 is added to Cb 

15 and Cr in Equation (6) so as to cause Cb and Cr to become 0 

^ Cb ^1 and 0^ Cr ^1. 

The parameters a and /3 in Equation (3) and parameters 
a and b in Equation (6) are recorded in the recording medium 
6, together with the output image data S2. 

20 The reading means 7 of the printer 2 reads out the output 

the image data S2 and the parameters a , /3 , a and b from the 
recording medium 6. The inverse processing means 8 carries 
out the inverse image processing , which is the inverse 
processing of the image processing carried out in the 

25 processing means 5 in the digital camera 1, on the output image 
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data S2. More specif ically, the component Cr and Cb are 
calculated by using the parameters a and b according to the 
following Equation (7): 
Cb=Cb'/a-0.5 
5 Cr=Cr'/b-0.5 (7) 

The signal values R2 , G2 , and B2 are found by solving Equation 
(5) and the signal value Rl is then found according to the 
following Equation (8): 

Rl = ( (R2 + 0.099)/1 .099) 1/0 * 45 (if R2^0.081) 

10 Rl=R2/4.5 (if R2<0.081) (8) 

Likewise, the signal values Gl and Bl are found according to 
Equation (8) replacing R with G or B. The signal values Rl , 
Gl, and Bl comprise the inverse-processed image data SI'. 

The inverse AWB processing means 9 obtains the inverse 
15 AWB image data SO ' by carrying out the inverse AWB processing, 
which is the inverse of the AWB processing carried out by the 
first AWB means 4, on the inverse-processed image data SI'. 
More specifically, processing shown by Equation (9) below 
using the parameters a and /3 is carried out on the signal 
20 values Rl , Gl and Bl comprising the inverse-processed image 
data SI' and the inverse AWB image data SO' comprising signal 
values R0 ' , GO' and BO' are obtained: 
R0 '=R1/ a 

G0'==G1 (9) 
25 B0'=B1//? 
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The inverse AWB image data SO' are not exactly the same as 
the original image data SO immediately after photographing , 
due to a calculation error or clipping. However, the inverse 
AWB image data SO' can be regarded as the original image data 
5 SO immediately after photographing. 

In the second AWB means 10, AWB processing is carried 
out on the inverse AWB image data SO' according to the method 
described in Japanese Patent Application No. 11 ( 1999 ) -70186 , 
for example. Hereinafter, the AWB processing carried out in 
10 the second AWB means 10 will be explained. The signal values 
R0 ' , GO ' and BO ' refer to tristimulus values R0 ' , G0 f and BO ' . 
First, the tristimulus values R0 ' , GO', and BO' comprising 
the inverse AWB image data SO' are converted into CIE1931XYZ 
tristimulus values and CIE1976uv chromaticity values 
15 according to the following Equations (11) and (12): 

X R0' 

Y = | A| • GO ' ( 11 ) 

Z BO ' 



20 U=4X/(X+15Y+3Z) 

v=9Y/ (X+15Y+3Z ) ( 12 ) 

where the matrix |a| is a matrix for converting the tristimulus 
values R0 ' , GO', and B0' into the tristimulus values X, Y, 
and Z, and the following numbers can be used, for example: 

25 0.4124 0.3576 0.1805 
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lAl = 0.2126 0.7152 0.0722 (13) 
0.0193 0.1192 0.9505 

The tristimulus values X, Y, and Z may be found by using 
a look-up table instead of the matrix I A I . 

Averages uO and vO of the chromaticity values u and v 
found by using Equation (12) are calculated according to 
Equation (14) below: 

u0= 2 u/n 

v0=2v/n (14) 
where n is the number of pixels. The averages uO and vO are 
converted into physiological primary colors L0, M0 and SO 
according to Equation (15)-(17) below: 
sx=9 . 0 XuO ( 6 . 0 Xu0-16 .0 XvO+12 .0) 
S y=4.0XvO(6.0XuO-16.0XvO + 12.0) (15) 

X0=sx/sy 

Y0=100.0 (16) 
Z0=( 1 . 0-sx-sy ) sy 

L0 X0 

M0 = |B| • Y0 (17) 
SO Z0 

where the matrix | B | is a matrix for converting the tristimulus 
values X0, Y0 and Z0 into the physiological primary colors 
M0, L0 and SO. The following matrix can be used as the matrix 



|B|: 

0.3897 0.6890 -0.0787 
|B| = -0.2298 1 . 1834 0 .0464 ( 18) 

0 0 1.0 

5 Furthermore, the tristimulus values X, Y, and Z found 

by using Equation (11) are converted into physiological 
primary colors L, M f and S according to Equation (17). 

The physiological primary colors L, M, and S are 
subjected to chromatic adaptation conversion according to 
10 Equations (19) and (20) , and converted physiological primary 
colors L ' , M' , and S' are obtained: 
L L 

M ' a | C | 'M (19) 
S ' S 
15 where (a -Ln+(l-a) -L0)/L0 

|C| = (a -Mn+(l-a) *M0 ) /M0 (20) 

(a -Sn+( 1-a) -SO ) /SO 
L0, M0 and SO are physiological primary colors of the 
chromaticity value averages found by using Equation (17) and 
20 regarded as a white color of a photographing light source used 
when obtaining the original image data SO. Ln, Mn, and Sn 
are chromaticit ies of a white color of a reference 
photographing light source upon obtaining the original image 
data SO. When Ln, Mn, and Sn are unknown, chromaticity (un, 
25 vn) of a neutral point equivalent to D50-D60 daylight can be 
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used for Ln, Mn, and Sn. a is a value indicating an adaptation 
state, and 1 is normally used for a. 

The converted physiological primary colors L', M ' and 
S' are converted into processed tristimulus values R3 , G3 , 
and B3 according to Equation ( 21 ) below and the processed image 
data S3 are obtained: 

R3 L ' 

G3 = | A | - 1 | B | ^ x -M' (21) 
B3 S ' 

An operation of this embodiment described above will 
be explained next. Figure 2 is a flow chart showing the 
operation of this embodiment. The imaging means 3 in the 
digital camera 1 carries out photographing of a scene to obtain 
the original image data SO (Step SI). The first AWB means 
4 carries out the first AWB processing according to Equations 
(2) and (3) on the original image data SO and the AWB image 
data Si are obtained (Step S2). In the processing means 5, 
the image processing comprising the scene density conversion 
processing and the chroma enhancement processing is carried 
out on the AWB image data Si according to Equations (4) to 
( 6 ) , and the output image data S2 are obtained ( Step S3 ) . The 
output image data S2 are recorded in the recording medium 6, 
together with the parameters a , /3 , a and b (Step S4 ) . The 
procedure described above is the procedure carried out in the 
digital camera 1. 



In the printer 2, reading means 7 reads the output image 

data S2 and the parameters a , ft , a and b from the recording 
medium 6 (Step S5). The inverse processing means 8 in the 
printer 2 carries out the inverse image processing, which is 
5 the inverse processing of the image processing carried out 
by the processing means 5 in the digital camera 1 , on the output 
image data S2 according to Equations (7) and (8) and the 
inverse-processed image data SI' are obtained (Step S6) . The 
inverse AWB means 9 then carries out the inverse AWB processing, 

10 which is the inverse processing of the AWB processing carried 
out by the first AWB means 4 in the digital camera 1, on the 
inverse-processed image data SI' according to Equation (9), 
and the inverse AWB image data SO' are obtained (Step S7). 
In the second AWB means 10, the second AWB processing is 

15 carried out on the inverse AWB image data SO' according to 
Equations (11) to (21), and the processed image data S3 are 
obtained (Step S8) . The processed image data S3 are subjected 
to other kinds of processing such as color conversion 
processing adequate for printing if necessary, and output by 

20 the output means 11 (Step S9) and the procedure is completed. 

As has been described above, in this embodiment, the 
inverse image processing, which is the inverse processing of 
the scene density conversion processing and the chroma 
enhancement processing carried out by the processing means 

25 5 , is carried out on the output image data S2 obtained by the 
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digital camera 1, and the inverse AWB image data SO' are 
obtained by carrying out the inverse AWB processing which is 
the inverse of the first AWB processing. The processed image 
data S3 obtained by carrying out the second AWB processing 
5 on the inverse AWB image data SO' can be regarded as image 
data having been subjected to color balance adjustment almost 
the same as color balance adjustment in the case of carrying 
out the second AWB processing directly on the original image 
data. Therefore, the processed image data S3 reflecting 

10 processing performance of the second AWB processing which is 
more sophisticated than the first AWB processing carried out 
in the digital camera 1 can be obtained. In this manner, a 
high quality print can be generated. 

In the above embodiment, the image processing comprising 

15 the scene density conversion processing and the chroma 
enhancement processing is carried out by the processing means 
5, in addition to the first AWB processing in the digital 
camera 1. However, some digital cameras do not comprise the 
processing means 5. In this case, image data output from the 

20 digital camera 1 are the AWB image data SI having been 
subjected to the first AWB processing. Therefore, in this 
case, the inverse AWB image data SO' can be obtained by 
carrying out the processing by the inverse AWB means 9 in the 
printer 2. A configuration of an image output system 

25 comprising the printer 2 for carrying out this kind of 
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processing is shown in Figure 3 as another embodiment. 

In the previous embodiment, the parameters a , /? , a and 

b are recorded in the recording medium 6, together with the 

output image data S2. However, the printer 2 may have the 
5 parameters a , j3 , a and b. In this manner, recording the 

parameters a , /? , a and b in the recording medium 6 becomes 

unnecessary in the digital camera 1 and photographing can be 

carried out faster. 

Furthermore, in the previous embodiment, the image 
10 processing apparatus of the present invention is installed 

in the printer. However, it is needless to say that the image 

processing apparatus of the present invention can be used as 

a single image processing apparatus. 

Moreover, in the previous embodiment , the AWB processing 
15 according to Equations (2) and ( 3 ) is carried out by the first 

AWB means 4 in the digital camera 1. However, the AWB 

processing is not limited to this processing. The AWB 

processing described in Japanese Patent Application No. 

11 ( 1999 ) -70186 is also carried out in the second AWB means 
20 10 in the printer 2. However, the AWB processing is not 

limited to this AWB processing and various kinds of AWB 

processing can be used. 

For example, the AWB processing may be carried out by 

the second AWB means 10 according to a method using a color 
25 adaptive prediction equation by von Kries. Consider the case 
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of observing, under a light source 2, an image photographed 
under a light source 1. Let tr istimulus values based on 
physiological primary colors of neutral points of the light 
source 1 and the light source 2 be L lw , M lw/ and S lw , and L 2w , 
5 M 2w , and S 2w/ respectively. The method converts LMS 
tristimulus values (L, M, S) 1 under the light source 1 before 
AWB processing into LMS tristimulus values (L, M, S) 2 of the 
image under the light source 2 after white balance adjustment, 
as shown by Equation (22) below: 







f L 2w/ L 1W 0 0 ^ 
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However, the method according to the von Kries : color 
adaptive prediction equation converts colors other than the 

15 white color, such as a complexion, into unnatural colors in 
the case of an image photographed under a specific light source 
such as fluorescent light or tungsten light, although the 
method can adequately adjust the colors other than the white 
color in the case of an image photographed in daylight. 

20 Therefore, the AWB processing may be carried out by using 

a matrix having non-zero values in non-diagonal elements for 
the conversion of the physiological primary colors, instead 
of using the matrix having non-zero values only in diagonal 
elements thereof as shown in Equation (22). Hereinafter, 

25 this method will be explained below. 
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It is generally known that spectral reflectivity of an 
object in nature can be represented by 3 main components. 
Therefore, spectral reflectivity of an object having been 
photographed can be expressed as a linear combination shown 
in Equation (23) below, with the 3 main components X 3 (A), 
X 2 (A), and X 3 (A) (A: spectral frequency): 



Here, a 3 , a 2 , and a 3 are weights for weighted addition of the 
main components. 

Let a spectral radiation distribution of the light 
source 1 be F(A) and let L-cone, M-cone and S-cone of the 

tristimulus values be 7( A ), m( A ), and 5(A). LMS 
tristimulus values L 3 , M lf and S 3 under the light source 1 
become as follows: 



X( A )=a 1 X 1 ( A )+a 2 X 2 ( A )+a 3 X 3 ( A ) 



(23) 



(24) 



L 1 "J vis F Wx(A)T(A)dA 

- 3 lJ VIS F W X 1 W 1 W dA +3 2 / VIS F W X 2 W 1 W dA 

+ a 3/ vis F W x 3 W1(A)dA 



(25) 



M 1 "/vis F Wx(A)m(A)dA 

" a l/ VIS F W X 1 W " W d A +3 2 / VIS F W X 2 W S W dA 

+ a 3 J vis F W x 3 WS(A)dA 



(26) 



S 1 -/vis F(A)x(A)i(A)dA 

" 3 lJ VIS F W X 1 W 1 W d A +a 2 J VIS F W X 2 W 1 W d A 

+ a 3/ vis F W x 3 (A) s (A) d A 
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The integrations in Equations (24) to (26) are integrations 
in a visible-light range. However, the integrations may be 
carried out in an all-spectral range. Since the spectral 
radiation distribution F( A) of the light source 1, the main 
components X x ( A ), X 2 ( A ) , and X 3 ( A ), and the spectral 
sensitivity /(A), m(A), and 5(A) of the physiological 
primary colors are known, define L u , M 1± , and S 1L as follows: 



(27) 



L t1 =/ VIS F(A)X, (A)1(A)dA 

M ii "/vis F W" i W dA ( is = 1 "- 3 > 

S 1i "/ VIS FWx i W i W dA 
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The LMS tristimulus values of the object having the spectral 
reflectivity X(A) seen under the light source 1 become as 
follows : 



(28) 



Hi 



M 



11 

11 
5 11 



M 



•12 
12 
'12 



M 



-13 
13 



'13 



\ 


f a1 ) 




a 2 


/ 





Likewise, the LMS tristimulus values of the object having the 
20 spectral reflectivity X( A ) seen under the light source 2 
become as follows: 



(29) 
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Eliminating a x , 


a 2 / 


and a 


3 from Equations 


(28) 


and 


(29), 


following Equation (30) 


can be 


obtained 


• 
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Therefore, conversion from the LMS tristimulus values (called 
y r here) under the light source 1 into the LMS tristimulus 
values (called y 2 here) under the light source 2 can be 
expressed as follows: 

(31) 



y 2 =P 2 Pr 1 y a 



where 
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In this manner, the conversion can be carried out by using 
20 the 3X3 matrix P2P1" 1 having non-zero non-diagonal elements. 

The AWB processing may be carried out in the following 
manner. Figure 4 is a block diagram showing an outline 
configuration of AWB means for carrying out the following AWB 
processing in the second AWB means 10. As shown in Figure 
25 4, this AWB means comprises neutral point calculating means 
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24 for calculating, from the inverse AWB image data SO', the 
tristimulus values L lw , M lw , and S lw based on physiological 
primary colors of a neutral point in an image represented by 
the inverse AWB image data SO', a database 25 storing 
tristimulus values of a neutral point of each of light sources 
(hereinafter called known light sources) and parameters 
(hereinafter called known light source parameters) for 
converting tristimulus values under each of the known light 
sources into tristimulus values under a light source for 
observing the processed image data S3, parameter generating 
means 26 for generating a processing parameter P for carrying 
out the AWB processing on the inverse AWB image data SO' based 
on the known light source parameters stored in the database 
25, and processing means 27 for obtaining the processed image 
data S3 by carrying out the AWB processing on the inverse AWB 
image data SO' based on the processing parameter P having been 
generated by the parameter generating means 26. 

The image data SO are obtained by photographing a scene 
with the digital camera 1 under a light source (light source 
1), such as an ordinary fluorescent lamp, whose parameters 
for converting the tristimulus values thereof into the 
tristimulus values under the light source for observation are 
not known. The light source under which the processed image 
data S3 are viewed is the light source 2. 

Since the inverse AWB image data SO' can be regarded 



as the image data obtained by the digital camera 1 , the inverse 
AWB image data SO' comprise 8-bit signal values of RGB colors 
according to the ITU Rec. 709 standard, and the signal values 
are represented by R0 ' , GO', and BO'. 

The neutral point calculating means 24 converts the 
color signal value R0 ' comprising the inverse AWB image data 
SO' into a signal value RIO normalized to 0-1 according to 
the following Equations (32) and (33) which are equivalent 
to Equation ( 8 ) : 

( 32 > R0" = R0'/255 

<33> R10-(^) 1/0 - 4 W,81) 

R10-R074.5 (R0"< 0.081) 

Likewise, signal values G10 and BIO are also found by the same 
operation described above. 

Averages Rll, Gil, and Bll of the signal values RIO, 
G10 , and BIO are then found according to the following Equation 
(34) using N as the number of all pixels in the image 
represented by the inverse AWB image data SO': 



_1 

( 34 ) N ^all pixels 

N ^ all pixels 

N Jallpixels 



R11-1 f R10 
N Jul 

G11 = A f G10 

B11=4 f • BIO 



The calculated averages Rll, Gil, and Bll are converted 



into CIE1931XYZ tristimulus values according to Equation (35) 
below: 
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The calculated tristimulus values are defined as tristimulus 
values X lw , Y lw , and Z lw of the neutral point in the image 
represented by the image data SO. 

The tristimulus values X lw , Y lw , and Z lw are converted into 
tristimulus values based on the physiological primary colors 
according to Equation (36) below: 



(36) 
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0.4002 0.7076 
-0.2263 1.1653 
0.0 0.0 



-0.0808\ 
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0.9182 7 
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The tristimulus values are then used as the tristimulus values 
L lw , M lw , and S lw based on the physiological primary colors of 
the neutral point in the image represented by the inverse AWB 

20 image data S0 f obtained from the image data SO generated by 
photographing under the light source 1. 

The database 25 stores the known light source parameters 
for converting the tristimulus values in each of the known 
light sources into the tristimulus values under the light 

2 5 source 2 . The known light source parameters is the 3X3 matrix 
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in Equation (31) above. The known light source parameters 
are stored in the database 25 in relation to the tristimulus 
values of the neutral points of the known light sources. 

Calculation of the known light source parameters will 
5 be explained next. Spectral intensity distributions of the 
known light sources and the light source 2 for observation 
are known. For example, in the case where the processed image 
data S3 are printed, the light source for observation is 
CIE-D50. In the case where the processed image data S3 are 

10 reproduced on a monitor, the light source for observation is 
CIE-D65. Therefore, according to Equations (23)~(31), a 3 
X3 matrix P^a" 1 for converting LMS tristimulus values in a 
known light source A into the LMS tristimulus values under 
the light source 2 can be found as shown in Equation ( 3 7 ) below: 

15 y 2 = P 2 P A " 1 y A (37) 

where y 2 is the LMS tristimulus values under the light source 
2 , y A is the LMS tristimulus values under the known light source 
A, P 2 is a 3 X 3 matrix for finding the LMS values of the object 
having the spectral reflectivity X( A ) observed under the 

20 light source 2, and P A is a 3X3 matrix for finding the LMS 
values of the object having the spectral reflectivity X(A) 
observed under the known light source A. The matrix P 2 P A ~ X is 
stored in the database 25 as the known light source parameters. 
The matrix P 2 and the matrix P A " L may be found separately so 

25 that the matrix PA" 1 is used as the known light source 



parameters . 

The LMS tristimulus values under the light source 2 and 
under the known light source A may be found by using spectral 
reflectivity data of multiple colors such as 1269 colors in 
the Munsell color chip so that the matrix for converting the 
tristimulus values y A into the tristimulus values y 2 by using 
the least square method or the like can be used as the known 
light source parameters. Since the matrix found in the above 
manner is equivalent to P 2 P A _1 , the matrix P 2 is found by using 
Equation (29) and the matrix P A _1 is also found by using the 
matrix P 2 . In this manner, the matrix P 2 and the matrix P A _1 
are found separately. 

The matrix can be determined so as to convert a white 
color of the known light source A into a white color of the 
light source 2. By using the spectral reflectivity data of 
the multiple colors such as those in the Munsell color chip, 
the LMS tristimulus values under the light source 2 and under 
the known light source A are found. The matrix for converting 
the tristimulus values y A into the tristimulus values y 2 are 
then found by using the least square method or the like. At 
this time, in order to convert the white color of the known 
light source A into the white color of the light source 2, 
matrices V and V are found by decomposing the matrix to be 
found into a matrix V based on the chromatic adaptive 
prediction equation by von Kries and into a matrix V which 
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is normal to the tristimulus values y A : 

y 2 = (V a +V a ')Ya (38) 
where V A 'y A = 0. 

In this case, P 2 P A ' 1= V A +V A ' . By determining the matrix 
so that the white color of the known light source A can be 
converted into the white color of the light source 2 , the white 
color under the known light source A can agree with the white 
color under the light source 2, which is more preferable. By 
finding the matrix P 2 according to Equation (29) and then by 
finding the matrix P A ~ X by using the matrix P 2 , the matrices 
P 2 and P A " X can be obtained separately. 

The parameter generating means 26 generates the 
processing parameter P for carrying out the color balance 
adjustment processing on the image data SO from the known light 
source parameters stored in the database 25, based on the 
tristimulus values L lw , M lw , and S lw of the neutral point of 
the light source 1 and the tristimulus values of the neutral 
points of the known light sources having the neutral points 
close to the neutral point of the light source 1 . Hereinafter, 
generation of the processing parameter P will be explained 
next . 

The parameter generating means 26 finds a distance in 
the LMS color space between the tristimulus values L lw , M lw/ 
and S lw of the neutral point input from the neutral point 
calculating means 24 and the tristimulus values of the neutral 



points of the known light sources stored in the database 25, 
and selects two of the known light sources having the distances 
shortest and second to the shortest. More specifically, the 
known light sources which are closer to the neutral point of 
5 the light source 1 in the LMS color space, that is, the known 
light sources which can be regarded as of the same type as 
the light source 1, are selected by plotting the tristimulus 
values of the neutral points of the known light sources stored 
in the database 25 and the tristimulus values L lw , M lw , and 

10 S lw in the LMS color space. Known light sources A and B are 
selected here, as shown in Figure 5. 

The known light source parameters of the known light 
source A are the matrix P^" 1 in Equation (37) and known light 
source parameters of the known light source B are PzPb" 1 in 

15 the following Equation (39): 
y 2 = P 2 Pb' 1 Yb (39) 
where y B is LMS tristimulus values under the known light source 
B and P B is a 3X3 matrix for finding the LMS values of the 
object having the spectral reflectivity x( A. ) observed under 

20 the light source B. 

Like Equation (38) above, the parameters of the known 
light source B can be found according to the following Equation 
(40): 

y 2 = (V B +V B ' )y B (40) 
2 5 where V B 'y B = 0. 
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Once the two known light sources A and B are selected 
in the above manner, an intersection point C between a line 
AB connecting the neutral points (points A and B) of the known 
light sources A and B and a perpendicular to the line AB from 
5 the neutral point of the light source 1 (point X in Figure 
5) is found in the LMS color space. If the intersection point 
C divides the line AB into a ratio of (1- a ): a , the 
intersection point C is equivalent to a neutral point of a 
light source having a spectral intensity distribution 
10 composed with the spectral intensity distributions of the 
known light sources A and B in a ratio of a z (1-a ) , as shown 
in Figure 6. 

After the intersection point C has been found in the 
above manner, parameters of the light source corresponding 
15 to the intersection point C are calculated by carrying out 
an interpolation operation on the known light source 
parameters of the known light sources A and B. 

More specifically, based on a calculated in the above 
manner, a matrix P c for converting the LMS tristimulus values 
20 under the light source corresponding to the intersection point 
C into the tristimulus values under the light source 2 is found 
according to Equation (41) below: 

P c = aP 2 P A ^+( 1-a )P 2 P B ~ 1 (41) 

In the case where the matrices P 2 , P*" 1 , and P B _1 are found 
25 separately, the matrix P c can be found according to the 
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following Equation (41'): 



P c = p 2 ( aP t +(l-a )P B ) 



(41' ) 



Therefore, LMS tristimulus values y c under the light 
source corresponding to the intersection point C can be 
converted into the LMS tristimulus values under the light 
source 2 according to the following Equation (42): 



Meanwhile, the intersection point C is on the line AB 
connecting the neutral points of the known light sources A 
and B in the LMS color space and do not agree with the light 
source 1. Therefore, in order to convert a white color under 
the light source 1 into the white color under the light source 
2, the matrix P c is corrected in the following manner. First, 
a correction matrix Ap is added to the matrix P c so that the 
white color in the light source 1 can be converted into the 
white color under the light source 2 according to the Equation 
(43) below: 



where L lw , M lw , and S lM are the tristimulus values based on the 
physiological primary colors of the neutral point of the light 
source 1 and L 2w , M 2w , and S 2tt are the tristimulus values based 
on the physiological primary colors of the neutral point of 



y2 = P c y c 



(42) 



(43) 




the light source 2. 

The matrix P c and Ap are decomposed into the component 
V according to the chromatic adaptive prediction equation of 
von Kries from the light source 1 to the light source 2 and 
other components Q c and A Q as shown by Equations (44) and (45) 
below: 

PC = QCV (44) 

AP = AQV (45) 
Since the tristimulus values L lw , M lw/ and S lw and the 
tristimulus values L 2w , M 2w , and S 2w have the following 
relationship shown by Equation (46) below, substituting 
Equation (43) with Equations (44) and (45) yields Equation 
(47) below: 



(46) 
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The following Equation (48) is obtained by transforming 
Equation ( 47 ) : 



(48) 
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where E is a unit matrix. 
From Equation (44), 
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(49) 



is obtained, meaning the right side of Equation (48) is known, 
Therefore, replace AQ in the following manner: 
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are obtained. If values of the vectors (qOO , qOl , q02 ) , (qlO, 
qll, ql2), and (q20, q21, q22) become minimum, components in 
the correction matrix AP also become minimum. Therefore, by 
adding a minor correction to the matrix P cf the neutral point 
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under the light source 1 can be converted into the neutral 
point under the light source 2. 

The values of the vectors (qOO, qOl, q02) / (qlO, qll, 
ql2 ) , and (q2 0 / q21 , q22) become minimum only in the case where 
5 these vectors are parallel to a vector (L 2w/ M 2w , S 2w ) . 
Therefore, by determining the vectors (qOO, qOl, q02 ) , (qlO, 
qll, ql2), and (q20, q21, q22) according to Equations 
(50)~(52) below, the matrix AQ can be found: 



(51) 

15 

(52) 




Therefore, the correction matrix A P can be found 
according to Equation (45) above. In this manner, a matrix 
20 M for converting the LMS tristimulus values under the light 
source 1 into the LMS tristimulus values under the light source 
2 can be found by the following Equation (53): 

M = P c + AP (53 ) 

The matrix M converts the tristimulus values of the 
25 neutral point under the light source 1 into the tristimulus 
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values of the neutral point under the light source 2. Since 
the matrix M has been obtained by correcting the matrix P c 
for converting the LMS tristimulus values under the light 
source corresponding to the intersection point C into the LMS 
tristimulus values under the light source 2, the matrix M also 
has a characteristic of converting the LMS tristimulus values 
under the light source 1 into the LMS tristimulus values under 
the light source 2. 

The matrix M having been generated in the above manner 
is input to the processing means 27 as the processing parameter 
p. in the processing means 27, the color signals RO ' , GO' 
and BO ' comprising the inverse AWB image data SO ' are converted 
into the signal values Rll, Gil, and Bll normalized according 
to Equations (33) and (34). The signal values Rll, Gil, and 
Bll are then converted into CIE1931XYZ tristimulus values Xll , 
Yll, and Zll according to Equation (54) below: 
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The tristimulus values Xll, Yll, and Zll are further 
converted into tristimulus values Lll, Mil and Sll based on 
physiological primary colors according to Equation (55) 
below: 
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The tristimulus values Lll, Mil, and Sll are converted 
by using the matrix M according to Equation (56) below, and 
converted tristimulus values L3, M3 , and S3 are obtained: 



/L3\ 
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By carrying out inverse processing of Equations (55) and (56) 
on the converted tristimulus values L3 , M3 and S3, the 
10 processed signals R3 , G3 , and B3 comprising the processed 
image data S3 are obtained. 

In addition, all of the contents of Japanese Patent 
Application Nos. 1 1 ( 1 9 9 9 ) -2 3 12 1 5 and 2000-173245 are 
incorporated into this specification by reference. 
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